Learning TRECVID'08 High-Level Features from YouTube

Run No. Run ID Run Description infMAP (%) training on TV08 data 1 IUPR-TV-M SIFT visual words with maximum entropy 6.1 2 IUPR-TV-MF SIFT with maximum entropy, fused with color+texture and motion (NN matching) 5.9 3 IUPR-TV-S SIFT visual words with SVMs 5.3 4 IUPR-TV-SF SIFT with SVMs, fused with color+texture and motion (NN matching) 6.3 training on YouTube data (no use of standard training sets) 5 IUPR-YOUTUBE-S SIFT visual words with SVMs 2.2 6 IUPR-YOUTUBE-M SIFT visual words with maximum entropy 2.1 We participated in TRECVID’s High-level Features task [17] to investigate online video as an alternative data source for concept detector training. Such video material is publicly available in large quantities from portals like YouTube. In our setup, tags provided by users during video upload serve as weak ground truth labels, such that thousands of concepts can be learned without manual annotation effort. On the downside, online video as a domain is complex, and the labels associated with it are coarse and unreliable, such that performance loss can be expected compared to high-quality standard training sets. To find out if it is possible to train concept detectors on web video, our TRECVID experiments compare state-of-the-art (visual only) concept detection systems when (1) training on the standard TRECVID development data and (2) training on clips downloaded from YouTube. Our key observation is that YouTube-based detectors work well for some concepts, but are overall significantly outperformed by the “specialized” systems trained on standard TRECVID’08 data (giving a infMAP of 2.2% and 2.1% compared to 5.3% and 6.1%). An in-depth analysis shows that a major reason for this seems to be redundancy in the TV08 dataset.